Skip to content

Opt-in first-recorded-failure ordering for all() with a per-run marker (#307) - #348

Merged
rmcdaniel merged 4 commits into
mainfrom
general/307-first-recorded-failure
Nov 19, 2026
Merged

rmcdaniel merged 4 commits into
mainfrom
general/307-first-recorded-failure

Conversation

@rmcdaniel

@rmcdaniel rmcdaniel commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

Related: #307, which stays open for the explicit propagate, cancel_remaining and settle policies (durable-workflow/.github#232). Follows #338/#341. The default flip is tracked in #351.

Semantics established

  • Embedded Workflow (main 67e8a969):
    • finishParallelDispatch raises a recorded group failure ahead of open members, and nothing is cancelled.
    • ParallelFailureSelector picks the earliest recorded_at in milliseconds, then the lower leaf index.
  • Native Server (main 52e3ec00, selection_resume): a failed member wakes the parent. A success wakes it only once the whole group has succeeded. Nothing is cancelled.
  • Contract (.github conformance/parallel-groups, lines 103 and 121): first failure in durable event order.
  • Python up to 2.19.1: raises the first failure early while a sibling is open (Never resend an open all() group; raise a member failure at once (#338) #341). Once every member has settled, it raises the lowest-position failure.

Change: per-run marker, two-phase rollout (pattern of workflow#1239)

  • Read always. A run that recorded the version marker durable_workflow.parallel_failure_order (v1), directly before its first all-group containing an activity or child, raises the first failure in durable event order, settled or not. It records no per-group decision, because history is append-only. Every worker of this release follows a recorded marker on tasks, queries and updates, whatever its setting.
  • Write opt-in. Worker(record_parallel_failure_order=True) or DURABLE_WORKFLOW_RECORD_PARALLEL_FAILURE_ORDER=true turns writing on. Any other value is refused when the Worker is created. replay() and Replayer.replay() take the same keyword. Default off: new runs send byte-identical commands to 2.20.0 (and 2.19.1).
  • Unmarked runs keep the released rule for life: lowest position once settled, and the all(): a failed member while siblings are pending blocks the run with 409 parallel_sequence_mismatch #338 decision when raised early. A run with a pending cancellation request at its first group records no marker.
  • Rollout: the docs and CHANGELOG say to enable writing only after every worker on the task queue runs this release. Published 2.20.0 and earlier leave a marked run replay-blocked until it is repaired. Turning recording off again is safe. Rolling back requires draining marked runs first. Record the parallel failure-order marker by default (phase 2 of #307) #351 flips the default in the next minor.

Live (Server built from main)

  • Native, default off: no marker, py307.fail#1. 2.19.1 also replays that history.
  • Native, env on: marker, py307.fail#3. 2.19.1 refuses that history.
PHP Server, review scenario 2.19.1 → this head
zero-command catch (await_result on a select handle) fail-m0 throughout
update answered by 2.19.1 between tasks update fail-m0, compensation fail-m0
update by 2.19.1 before this head's main task fail-m0 / fail-m0
new run, recording on, with update fail-m1, update fail-m1

Fixtures (no existing fixture changed or retired)

  • Recorded by published 2.19.1, replaying the released rule:
    • zero-command catch, both mid-run and complete;
    • update between tasks;
    • update before the next worker's main task;
    • select cancel;
    • native settled group, complete and in flight.
  • Recorded with the marker, replaying the new rule without the setting:
    • zero-command catch, both mid-run and complete;
    • new run with an update;
    • a native new run.
  • First task of a run, from the same Server history:
    • failure-order-opt-in-first-task-records-marker (recording on);
    • failure-order-default-first-task-sends-released-commands (recording off);
    • parallel-failure-order-opt-in-new-run-emission.
  • parallel-group-metadata-emission, the two completion-limit fixtures and business-clock-parallel-failure-selection-gap are unchanged and pass.

Tests

  • tests/test_parallel_failure_order.py:
    • default off emits no marker;
    • opt-in on writes it once, directly before the first group that can fail, and not before timer-only groups;
    • marked and unmarked histories replay the same with the setting on or off;
    • an update in an unmarked run reports the released rule;
    • the environment variable is parsed, and malformed values are refused;
    • the worker writes the marker only when enabled.

Related

Checks

  • Locally:
    • non-integration suite: 7772 passed;
    • ruff and mypy pass;
    • corpus validator passes;
    • integration suite against a Server stack built from server main: 75 passed, 57 skipped (run at the previous head).

🤖 Generated with Claude Code

When several members of an all() group had failed by the time a task
reached it and every member had settled, the worker raised the failure
with the lowest input position. Workflow's embedded engine raises the
failure recorded first, so the exception a workflow caught depended on
whether its task ran before or after the other members finished.

A task reaching a settled group now raises its first recorded failure.
It records the member it raised with the #338 decision side effect only
when that member is not also the lowest-position failure, so its
commands are unchanged whenever both rules agree. Replay raises the
member a decision names. A settled group with no decision after which
the run closed or did further durable work came from a released worker
and keeps the position rule and its clock behavior.

Native Server fixtures recorded by published 2.19.1 (position rule,
complete and in flight) and by this version (decision, deciding task)
cover both rules. The selection-gap fixture's frontier expectation is
retired as same-protocol evidence under a corrected fixture.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Review of the previous head showed that deciding from history alone is
unsound. A released worker could catch a settled group's lowest-position
failure and finish the task with no command, for example while awaiting a
select() handle. A later task then saw no decision and nothing after the
group, took it for a new frontier and chose another member. An update
answered in between had already reported the first choice.

Each run now records the version marker
durable_workflow.parallel_failure_order once, directly before the first
all-group that contains an activity or child workflow, when that group is
first scheduled. A marked run raises the first failure in durable event
order, whether or not the other members have settled, and needs no
per-group decision because history is append-only. A run whose first such
group was recorded without the marker keeps the released rule (position
when settled, the #338 decision when raised early) for its whole life, on
every task, query and update. A run with a pending cancellation request at
that point records no marker.

Native and PHP Server built from main accept the marker. Published 2.19.1
refuses a marked run as nondeterministic rather than choosing differently.
Fixtures recorded by 2.19.1 (zero-command catch, update before and between
this version's tasks, select() cancellation, settled group in flight)
replay under the released rule. Fixtures recorded by this version replay
under the new rule. The selection-gap fixture is kept as released
behaviour. Three fixtures of a new run's first group emission are
superseded because they now include the marker.

Docs now say Python follows the contract's event order where embedded
Workflow compares millisecond timestamps, then index
(durable-workflow/.github#415). The select() base-sequence bug found in
review is #350.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@rmcdaniel rmcdaniel changed the title Raise a settled all() group's first recorded failure (#307) Gate all() first-recorded-failure selection on a per-run marker (#307) Nov 19, 2026
Published 2.19.1 cannot replay a run that recorded the parallel
failure-order marker. The Server marks the workflow task replay-blocked
with no retry, and only a repair resumes it, which can hand it to an old
worker again. Rolling back would do the same to every marked run.

Follow the two-phase pattern of workflow#1239:

- Every worker reads and follows a recorded marker, on tasks, queries and
  updates.
- Writing it is off by default. Worker(record_parallel_failure_order=True)
  or DURABLE_WORKFLOW_RECORD_PARALLEL_FAILURE_ORDER=true turns it on. Any
  other environment value is refused when the Worker is created. replay()
  and Replayer.replay() take the same keyword.
- With the default, new runs send exactly the commands 2.19.1 sends.
- The next minor release flips the default (#351).

This removes the same-protocol supersession of new-run emission fixtures:
parallel-group-metadata-emission and the two completion-limit fixtures
are unchanged and pass again. New fixtures cover the first task with
recording on and off. The marked native and PHP histories replay under
the new rule without the setting.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@rmcdaniel rmcdaniel changed the title Gate all() first-recorded-failure selection on a per-run marker (#307) Opt-in first-recorded-failure ordering for all() with a per-run marker (#307) Nov 19, 2026
Move the failure-order entry to Unreleased. Name 2.20.0 as the last
release that cannot replay a marked run.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@sonarqubecloud

Copy link
Copy Markdown

@rmcdaniel
rmcdaniel merged commit 0d5283c into main Nov 19, 2026
25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant